Papers with average F1 score
MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension (D19-58)
Copied to clipboard
| Challenge: | MRQA datasets have been used to benchmark progress in general-purpose language understanding. |
| Approach: | They propose to combine 18 question answering datasets into one shared task to evaluate their generalization capabilities. |
| Outcome: | The best system achieved an average F1 score of 72.5 on the 12 held-out datasets, 10.7 absolute points higher than baseline based on BERT. |
DeepGeneMD: A Joint Deep Learning Model for Extracting Gene Mutation-Disease Knowledge from PubMed Literature (D19-57)
Copied to clipboard
| Challenge: | Identifying and understanding the pathogenesis of genetic diseases is an essential task. |
| Approach: | They propose a joint deep learning model for gene mutation-disease knowledge extraction that adapts the state-of-the-art hierarchical multi-task learning framework for joint inference on named entity recognition and relation extraction. |
| Outcome: | The proposed model achieves the average score of 0.45 on recognizing gene activities and disease entities and the average F1 score of 0.3 on extracting relations, ranking 1st in the AGAC RE task. |
CLER: Cross-task Learning with Expert Representation to Generalize Reading and Understanding (D19-58)
Copied to clipboard
| Challenge: | In-domain datasets are used to train and validate our model, and other out-of-domain data are used for validation. |
| Approach: | They propose a model which uses cross-task learning with expert representation for the generalization of reading and understanding. |
| Outcome: | The proposed model achieved an average F1 score of 66.1 % in the out-of-domain setting, which is a 4.3 percentage point improvement over the official BERT baseline model. |
Generalizing Question Answering System with Pre-trained Language Model Fine-tuning (D19-58)
Copied to clipboard
| Challenge: | Existing methods focus on improving in-domain performance, leaving open the question of how they can generalize to out-of-domain and unseen RC tasks. |
| Approach: | They propose a multi-task learning framework that learns the shared representation across different tasks and builds on a large pre-trained language model and fine-tuned on multiple RC datasets. |
| Outcome: | The proposed framework improves the BERT-Large baseline by 8.39 and 7.22 respectively. |
Sanaphor++: Combining Deep Neural Networks with Semantics for Coreference Resolution (L18-1)
Copied to clipboard
Julien Plu, Roman Prokofyev, Alberto Tonon, Philippe Cudré-Mauroux, Djellel Eddine Difallah, Raphaël Troncy, Giuseppe Rizzo
| Challenge: | Coreference resolution is a challenging task in Natural Language Processing . since a few years, the biggest step forward has been made using deep neural networks . |
| Approach: | They propose to improve coreference resolution by adding semantic features to a top-level deep neural network system . they evaluate a shared task dataset and compare it to the state-of-the-art system based on Stanford deep-coref . |
| Outcome: | The proposed system achieves 1.13% gain over the CoNLL 2012 dataset and the state-of-the-art system. |
Detecting Narrative Elements in Informational Text (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent work has focused on identifying narrative elements in personal stories texts, but this paper focuses on informational texts. |
| Approach: | They propose a novel NLP task for detecting narrative elements in raw text by adapting elements from the oral narrative theory of Labov and Waletzky and adding a new narrative element of their own. |
| Outcome: | The proposed scheme achieves an average F1 score of 0.77 and is better suited for informational texts than the oral narrative theory. |
TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for Relation Extraction (RE) still show a high error rate . label errors account for 8% absolute F1 test error, and more than 50% of examples need to be relabeled. |
| Approach: | They validate the most challenging 5K examples using trained annotators and analyze misclassifications on the challenging instances. |
| Outcome: | The proposed methods perform well on the most challenging datasets and improve on the relabeled test set. |
TechING: Towards Real World Technical Image Understanding via VLMs (2026.findings-eacl)
Copied to clipboard
| Challenge: | Modern day vision language models struggle when it comes to understanding technical diagrams . a large synthetically generated corpus is needed to train and evaluate VLMs on hand-drawn images . |
| Approach: | They propose a large synthetically generated corpus for training VLMs and evaluate them on hand-drawn images. |
| Outcome: | The proposed model improves ROUGE-L performance of Llama 3.2 11B-instruct by 2.14x on synthetic images on real-world images. |
A Streamlined Span-based Factorization Method for Few Shot Named Entity Recognition (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing approaches to few-shot named entity recognition require large amounts of labeled data. |
| Approach: | They propose a streamlined span-based factorization method that solves few-shot NER problem . they propose to decompose the span-level alignment problem into several refined procedures . |
| Outcome: | The proposed method achieves an average F1 score improvement of 12 points on the FewNERD dataset and 10 points on SNIPS dataset. |
A Girl Has A Name: Detecting Authorship Obfuscation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing authorship attribution methods are not stealthy as they degrade text smoothness in detectable manner. |
| Approach: | They evaluate the stealthiness of authorship attribution methods under an adversarial threat model and show that they are not stealthy . |
| Outcome: | The proposed methods can be identified with an average F1 score of 0.87 . |
PsyCoT: Psychological Questionnaire as Powerful Chain-of-Thought for Personality Detection (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have demonstrated remarkable zero-shot performance across various NLP tasks. |
| Approach: | They propose a method which mimics the way individuals complete psychological questionnaires in a multi-turn dialogue manner and prompts an LLM to rate individual items at each turn. |
| Outcome: | The proposed method improves the performance and robustness of the standard GPT-3.5 personality detection task on two benchmark datasets. |
QueryForm: A Simple Zero-shot Form Entity Query Framework (2023.findings-acl)
Copied to clipboard
Zifeng Wang, Zizhao Zhang, Jacob Devlin, Chen-Yu Lee, Guolong Su, Hao Zhang, Jennifer Dy, Vincent Perot, Tomas Pfister
| Challenge: | Form-like document understanding is a key yet under-investigated problem . endlessly training specialized models on new document types is not scalable in many practical scenarios. |
| Approach: | They propose to use large-scale query-entity pairs generated from form-like webpages to pre-train QueryForm. |
| Outcome: | The proposed framework sets state-of-the-art average F1 score on XFUND and Payment benchmarks. |
A Novel Challenge Set for Hebrew Morphological Disambiguation and Diacritics Restoration (2020.findings-emnlp)
Copied to clipboard
| Challenge: | morphological parsers face a formidable challenge with unbalanced ambiguities in homographs . case of unbalanciated ambiguity is difficult to disambiguate, especially in cases of unbalancing . a new dataset improves the overall average F1 score for Hebrew homograph . |
| Approach: | They propose a challenge set for Hebrew homographs with substantial attestation of each analysis of 21 Hebrew homographies. |
| Outcome: | The proposed set improves the average F1 score for Hebrew homographs by 0.67 . the annotated datasets are made publicly available for further research. |
Inter-Passage Verification for Multi-evidence Multi-answer QA (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing multi-answer question answering systems struggle to retrieve and synthesize a large number of evidence passages. |
| Approach: | They propose a multi-answer question answering framework that generates a large set of passages and then processes each passage individually to generate an initial high-recall but noisy answer set. |
| Outcome: | The proposed framework outperforms baselines on the QAMPARI and RoMQA datasets, achieving an average F1 score improvement of 11.17%. |
Automatic Creation of Named Entity Recognition Datasets by Querying Phrase Representations (2023.acl-long)
Copied to clipboard
| Challenge: | Named entity recognition models rely on domain-specific dictionaries provided by experts . however, such dictionary sets are infeasible in many domains where they do not exist . |
| Approach: | They propose a framework that generates NER datasets with high-coverage pseudo-dictionaries . phrase retrieval models are used to retrieve popular entities rather than rare ones . |
| Outcome: | The proposed framework outperforms the previous best model by an average F1 score of 4.7 across five NER benchmark datasets. |
Explicit and Implicit Data Augmentation for Social Event Detection (2025.acl-long)
Copied to clipboard
| Challenge: | Social event detection relies on labeled data, but annotation is costly and labor-intensive. |
| Approach: | They propose a plug-and-play dual augmentation framework that combines explicit text-based and implicit feature-space augmentation to enhance data diversity and model robustness. |
| Outcome: | The proposed framework outperforms the best baseline model by 17.67% on the Twitter2012 dataset and 15.57% on the twitter2018 dataset in terms of the average F1 score. |
A Herd of Language Models Makes a Better Zero-shot Annotator for Clinical Named Entity Recognition (2026.findings-acl)
Copied to clipboard
| Challenge: | Clinical named entity recognition (NER) is a core task in clinical NLP. |
| Approach: | They propose a label-modeling method for M**ulti-LLM **A**nnotation using **R**epresentation learning to capture contextual similarity. |
| Outcome: | The proposed method improves the average F1 score by 8.6% over zero-shot baselines while reducing annotation costs. |
Towards Open Domain Event Trigger Identification using Adversarial Domain Adaptation (2020.acl-main)
Copied to clipboard
| Challenge: | supervised event trigger identification models can generalize better across domains . prior work focused on annotating specific categories of events or narratives from specific domains. |
| Approach: | They propose to use adversarial domain adaptation framework to build supervised event trigger identification models which can generalize better across domains. |
| Outcome: | The proposed model improves on literature and news domains with no labeled data. |
Unsupervised Cross-lingual Representation Learning at Scale (2020.acl-main)
Copied to clipboard
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, Veselin Stoyanov
| Challenge: | Pretraining multilingual language models at scale leads to performance gains for cross-lingual transfer tasks. |
| Approach: | They present a transformer-based multilingual masked language model pre-trained on 100 languages . they show that pretraining multilingual models at scale leads to significant performance gains . |
| Outcome: | The proposed model outperforms multilingual BERT (mBERT) on cross-lingual benchmarks. |
Zero-Shot Cross-Lingual NER Using Phonemic Representations for Low-Resource Languages (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing zero-shot cross-lingual NER approaches require substantial prior knowledge of the target language, which is impractical for low-resource languages. |
| Approach: | They propose a phonemic representation based on the International Phonetic Alphabet (IPA) to bridge the gap between representations of different languages. |
| Outcome: | The proposed method outperforms baseline models in low-resource languages with highest average F1 score and lowest standard deviation. |
When LLMs Read Tables Carelessly: Measuring and Reducing Data Referencing Errors (2026.acl-long)
Copied to clipboard
Yuqing Yang, Qi Zhu, Zhen Han, Boran Han, Zhengyuan Shen, Shuai Wang, Vassilis N. Ioannidis, Huzefa Rangwala
| Challenge: | Large language models (LLMs) perform well on table tasks, but they still make data referencing errors (DREs) prior studies have only offered limited, small-scale analyses. |
| Approach: | They propose inference-time strategies and lightweight critics to mitigate data referencing errors. |
| Outcome: | The proposed model achieves an average F1 score of 78.2% in detecting both in-distribution and out-of-difference DREs and assists inference for larger models. |
Can Large Language Models Infer Causal Relationships from Real-World Text? (2026.acl-long)
Copied to clipboard
| Challenge: | Existing work evaluating large language models relies on synthetic or simplified texts with explicit causal relationships. |
| Approach: | They develop a benchmark to evaluate LLMs' ability to infer causal relationships from texts . they use a dataset of texts with different levels of explicitness and complexity . |
| Outcome: | The proposed benchmark is the first-ever real-world dataset for this task. |
CARO: Chain-of-Analogy Reasoning Optimization for Robust Content Moderation (2026.findings-acl)
Copied to clipboard
| Challenge: | Current large language models struggle with ambiguous content moderation cases due to misleading "decision shortcuts" . authors propose a two-stage training framework to induce robust analogical reasoning in LLMs . |
| Approach: | They propose a two-stage training framework to induce robust analogical reasoning in LLMs . they bootstrap analogy reasoning chains via retrieval-augmented generation and SFT . |
| Outcome: | The proposed framework outperforms state-of-the-art reasoning models and specialized moderation models on ambiguous moderation benchmarks. |
Zero-Shot Cross-Lingual Document-Level Event Causality Identification with Heterogeneous Graph Contrastive Transfer Learning (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies focus on sentence-level ECI with high-resource languages, leaving document-level DECI with low-resourced languages under-explored. |
| Approach: | They propose a Heterogeneous Graph Interaction Model with Multi-granularity Contrastive Transfer Learning for zero-shot cross-lingual ECI. |
| Outcome: | The proposed model outperforms the state-of-the-art model on monolingual and multilingual scenarios by 9.4% and 8.2% of average F1 score. |